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NETWORK LEARNING 


1- INTRODUCTION 


In the early days of Artificial Intelligence, in the 90's, the two goals: developing a theory of intelligence and 
adaptiveness, and designing systems exhibiting intelligent behavior, .vere not only considered as the two 
sides of the same coin, but, truly, as the same problem. Turing developed his reflexions on intelligence, 
means to produce it and test i (his now famous test) [65], and less well-known but equally important to him, 
his research on morphogenesis and self-organization in the development of complex forms [64]. About the 
same time, Ashby "attempted to deduce what is necessary, what properties the nervous system must have if 
it is to behave at once mechanistically and adaptively " [4]. J. von Neumann proposed, through his theory of 
automata, to study the logical differences between “natural” and “artificial” automata. Others, building over 
the concepts of “forma! neurons” as introduced by Mc Culloch and Pitts [40], conceived adaptive systems 
that were shown to be capable of learning. Widrow [67] introduced the Adaline and Rosenblatt [51] the 
perceptron, further studied by Minsky and Papert [43]. Newell and Simon on their side advocated general 
means-ends analysis. 

However, with the mid-sixties, came disenchantment with these "universal" approaches. Failures, such as 
performing automatic transtation, or limitations clearly demonstrated, for instance on the perceptron [43], 
caused a shift of interest from general theories of intelligence more towards engineering and operational 
aspects. At the same time, a new stance evolved toward symbol level machines and knowledge intensive 
systems. This trend was particularly strong during the seventies. 

But its own limits in turn came under light. It is indeed nice to design more powertu! “intelligent” systems, but 
since this requires to include more knowledge, general as well as domain dependent, it follows that more 
sophisticated knowledge acquisition systems are needed. This is why we now watch a resurgent interest for 
efficient learning strategies and correlatively a return to more general and basic concern on the nature of 
intelligence and knowledge. 

in this paper, we will show how the original approach of the 4950's recently regained activity and proved quite 
efficient in a variety of learning situations. This rebirth of the adaptive systems approach -coined as 
“connectionism’- fies on ideas stemming from neurobiology: understanding the way human brain works 
might help us build intelligent machines, and has been justified by recent advances on learning algorithms in 


neural networks. The main arguments go as follows: 


1-1 Knowledge representation: in Artificial Intelligence, knowledge is usually represented in a local 
way: one concept, one atom of knowledge corresponds to one logical rule (e.g. in Prolog), one slot 
ina frame ... In brain theory, biologists have long held such local representation schemes: in the 
“grand-mother" cell model, it was postulated that each piece of knowledge, memory or concept had its 
associated neuron, responsible for its possible evocation. When | see my grand-mother, one of my 


Z 


neuron “recognizes” her and fires. But recent biological evidence, for example the fact that thousands 
of our neurons die out each day, imply that knowledge be most probably encoded in a distributed 
fashion in our brain: it is the coordinated pattern of activity of many neurons which really “codes” for a 
given piece of knowledge. In this way, many neurons can be used to memorize very different things, 
this feature will help provide fault tolerance. 

Knowledge representation systems in Al are often based on Logics. Of course, formal predicates are 
not what our brain is used to manipulate: it usually gets low level informations from its environment in 
the form of pieces of sound or light... and from these informations builds up abstract concepts and 
symbolic representations that may be used by upper level inference mechanisms. This process might 
involve intuition, analogy, non monotony and so on, which are hard to frame into logics-based 
formalism (this is actually an active research area in Al). An intelligent system should thus be capable of 
using different levels of representation and manipulation of data: we will show that connectionist 
systems usually rely on numerical data representation (“sub symbolic”) but are also capable of 


generating higher level representations at the symbolic level. 


4-2 Inference mechanisms in Al range from formal logical inference (e.g. in Prolog) to inheritance or 
markers passing techniques. In the brain, as far as is known today, “inferences” can only be made on 
the basis of propagation of activation through the neurons. We will see that connectionist networks 
also use this mechanism. 

4-3 Parallelism: almost any computer can compute faster than we do (a performance that we would 
describe as “stupid number crunching”). But almost all of us are better than any computer at 
understanding natural language, recognizing images, solving puzzles ... all activities that we would 
certainly call intelligent. This is certainly due to the fact that our brain makes heavy use of parallel 
processing: many different processes are running when we listen to a speaker for example. Many Al 
systems today try to implement such coordination between processing levels. However this is usually 
done mostly sequentially. Connectionist networks allow to fully exploit paratlelism, provided that 
adequate machines exist: but the on-going development of dedicated hardware (either digital, 


analogic, or optical) makes it possible today. 


1-4 Learning: in the brain, learning seems to take place at the level of synapses, which are the 
locations where neurons come into “contact” through neuro-transmitters. Those connections which 
are useful -for a given task- are reinforced through learning and the others progressively degenerate. 
In connectionist systems, learning takes place at the level of interactions between individual 


processing elements. 


These different remarks are at the root of connectionism and we will show in the following how they were 
embedded in the models. However, these models usually do not pretend to mimick the brain. Although 


3 


some researchers use them to modelize neuro biological or psychological phenomena, many others merely 
intend to provide new aigorithms to achieve tasks already implemented on sequential computers. We focus 
in this paper on the possible use of such models in Artificial Intelligence where examples have been 
developed in vision [11, 17, 55, 69], speech [8,14, 24, 48, 59], signal processing [23, 62], natural language 
[66], automatic diagnosis [37], associative memory [19, 32, 36], hardware design or combinatorial 


optimization [13, 30}. 


This paper overlaps somewhat with the presentation given by Hinton [26], in this volume. To usefully 
complement it, we have tried in this chapter to focus on the important problem of generalization, choosing to 


discuss that through various examples. 


The paper is organized as follows: in section 2, we present the paradigm of network learning, with some 
connectionist models for knowledge representation. In section 3 we present the notion of automata, 
section 4 is mostly devoted to learning on multi-layered networks by the algorithm of gradient back 
propagation and section 5 toa discussion of the generalization issues in mutt-layered networks through 


examples of implementation. 


Note to the reader: this paper is intended to show the possibility of directly using connectionist models 
within classical symbolic Machine Learning systems. The reader interested to get the back grounds and the 
deep foundations of the models could read [18,20] where we describe in full details the mathematical tools, 


the first adaptive models and the first linear learning models. 
2- NETWORK LEARNING 


2-1 Intelligent systems 


it is widely accepted today in the Artificial Intelligence community that any general intelligent system should 
include a general learning mechanism. For example, Laird & al [34], on their SOAR system, state that: « a 
system that is to be capable of general intelligence ... is to be able to: (1) work on the full range of tasks...(2) 
employ the full range of problem solving methods and representations... (3) learn about all aspects of the 
tasks...». They think that such an intelligent system should be based on some fundamental hypotheses, that 
are «almost universally accepted» within the Al community, namely: 
« - Physical symbo! system hypothesis: a general intelligence must be realized with a symbolic system. 

- Goal structure hypothesis: control in a general intelligence is maintained by a symbolic goal system ... 

- Production system hypothesis: production systems are the appropriate organization for encoding all 


long-term knowledge.» 


4 


Neural networks have recently [25, 53] been proposed as a new possibility to implement intelligent systems. 
Although they almost completely violate the three previous hypotheses, they are capable of solving many 
different tasks and learning about many aspects of these tasks. They are based on fully non symbolic 
knowledge representation and inference schemes: they use numerical representations and dynamical 
inference. Viewed this way, network models are complete foreigners within the Al systems world. This was 
most clearly the position of the early adaptive models developed in the 60s, such as the perceptron [43]. 
However, learning mechanisms [36, 43, 52] proposed recently lead to modify this view. indeed it is possible 
today to train networks through examples in such a way that, during the leaming process, they progressively 
build up a functional structure corresponding to the use of a set of symbolic descriptors, characteristic of the 
task. These descriptors constitute the "long-term memory” of the network. Although represented in the 
network in numerical form, they provide, when deciphered by the experts, a high-level representation of the 
problem. The emergence of this representation is the key to a better understanding of critical phenomena to 


Al, such as: 


- improvement with experience 
- transfer to related tasks 
- creation of rules 


- generalization... 
These abilities would make the network learning approach useful for the Al tool-box. 


The purpose of this paper is to show how these network models work and how they could be used by the 
Machine Learning community. Since these models have been almost completely developed outside of the 
At field, they make use of different vocabulary and concepts and their link with Al is still quite unclear. We will 
thus be able here to merely point at some aspects which seem to be important today, but we want to stress 
that much work remains to be done to elucidate the precise possible cross-fertilization between Ai and the 


connectionist approach. 


After a brief comparison of the Network Learning and traditional Machine Learning approaches, the following 
section will present the knowledge representation and inference mechanisms used in connectionist 


models, and later introduce the most classical network learning mechanisms. 
2-2 Knowledge representation on networks 


Classicaly, Al systems use knowledge representations based on logics, structured objects (frames, scripts...) 
or semantic networks. In ail these techniques, one concept is associated to one element in the system 
(one rule, or one frame or one node), i.e. the representation is local. Some early connectionist systems 


started by using local representations very similar to semantic networks. However, the “inference” 
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mechanism in these early models was not based on inheritance, but on a technique of propagation of activity, 
somewhat similar to a markers passing system [15]. A further difference between the traditional Al approach 
and connectionism resides in the modality of the contro! process. Classical Al needs a global supervisor 


instead of the distributed control employed in neural nets. 


2-2-1 Local representation 


This approach can be illustrated by an example taken from Waltz [66] in natural language disambiguation. The 
sentence: . “THE ASTRONOMER MARRIED A STAR” 
can be interpreted in two different ways (at least) depending on the particular meaning of the word “star”. 


LZ. 1: syntactic level 


ra [ster It: input level 


oon eee eee eee ee eee eee ee SS eee eee — ae a ee ae a oe o-oo 


celestial Laamet 
pee ligute Il:lexical level 











movie star 
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study (0 marry | IY: contextual level 


FIGURE 1: network for disambiguation [66] 
The figure shows a network used to interpret the sentence “THE ASTRONOMER MARRIED A STAR”. The sentence is 


analized at different levels (syntactic, input, lexical and contextual) that interact in a dynamical process of activation 
propagation. The different elements of knowledge in the problem are represented in a local way by cells in the network. 
Each piece of information is connected to others in the network by means of links or connections which are either 
reinforcing or inhibiting. Cells connected through reinforcing links will cooperate to support the same interpretation, 


whereas cells connected through inhibitory links (circled) will compete for different interpretations. 


A network is built with cells, organized in different levels (fig. 1): the syntactic level where the sentence is 
parsed, the input level, where it is entered into the network, the lexical level where the different meanings of 
the words are represented and the contextual level which sets the context into which the sentence is to be 
analyzed. This last level is supposed to have been activated by previous sentences entered in the 
processing of a larger text. Each level is composed of various cells which are using a saturation automaton 
rule as in equation (10) - see section 3 below-, with the activity levels kept in [0,1]. Celis are connected 
through two different sorts of connections: excitatory connections (like between “ASTRONOMY” and 


ee see oe Pee ore ra —_——- 
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Each time an example is presented at the input level, the hidden layer computes an activity pattern which is 
then processed at the output level, and produces the output of the network (the identification of the 


consonant). 


The input and output are represented in a local way. The input is just a “picture” of the FFT of the signal for 
the various sounds “ba”, ..., “gu”: at the input level, each automaton gives the spectral magnitude 
corresponding to a given frequency and time (there are 16 frequency levels and 20 successive time slices: 
fig. 3). At the output level, they use a coding by “position”: 3 bits are used to code the 3 consonants, “b” is 
coded as 100, "d” as 010 and “g” as 001. In other words, to each consonant is associated an automaton at 
the output level. The hidden units instead use a distributed representation shown in figure 3: consonant “b” 
is represented by the pattern of activity 10 1 of the 3 hidden units. However for some of the “bu” inputs, the 
pattern of activity is 1 1 1. Consonant “d” is always coded as pattern 0 0 0 by the hidden units, where 
consonant “g” is coded as 0 1 0. These hidden units can thus be viewed as feature extractors adapted to the 


peculiar task the network was trained for. 


Note that in this example, the inference is drawn by just propagating the activity of the input elements 
towards the output layer, where the “answer” of the network (here the consonant identity) is read. This is just 
a “one shot” process in contrast with the Walz's example described before where the inference was drawn 
from the long-term behavior of the network. Remark also that the distributed representation shown in figure 3 
is not so easy to interpret: the hidden cells seem to act as feature detectors for some characteristics of the 
inputs. Local representations are often easier to interpret, but for such models the generalization 
performances are not as good as for systems using distributed representations. We shall come back iater on 


this point in section 5. 
2-3 Various learning techniques on networks 


Different techniques are used for learning on networks, depending on how much knowledge is included 
into the network during the learning process. Some of them (non-supervised learning) can be viewed as 
sort of clustering methods: classes are progressively built, based on non conceptual descriptors, the 
representative of a class being a rea! vector (the weight). Examples of non supervised learning by networks 
are for example competitive learning [54], Kohonen's “topological maps" method [32], or Grossberg's 
Adaptive Resonance system [9]. 

In reinforcement models [5], the network is trained through examples and is progressively adapted so that its 
output maximizes the expected reinforcement [68]. In the fully supervised model, the network is provided 
with examples and associated desired answers. See [18] or Hinton [26] in this volume for more details on 
these different models. In this paper, we will focus on the supervised learning model, which we briefly 


present here. 


9 


The method is used whenever the problem can be described as one of associating with a given set of 
examples a corresponding set of associated responses. Examples are coded into the input layer of an 
automata network (see section 3). The connection weights of the network are then progressively adapted so 
that the desired response be identical -or as close as possible to identical- to the output of the network 
(fig.4). This requires that all states of the output layer automata be specified by the desired response state, 


whereas in the case of the reinforcement procedure, only a global signal is necessary. 


$tiittstH desired output 
+++244¢ + computed output 


poaq0 ©) 6.900 NETWORK 


FIGURE 4: supervised Jearning 


The figure shows a network used for supervised learning: the input pattern is presented to the network, the activity 
propagates through the network which computes the cells states on the output lavel. This computed output is then 


compared to the desired response and eventually the error is propagated through the network to modify the connection 


weights. 


Many problems can be set into this framework, we will present some of them in more details in section 5. 


Applications developed today are - 


- classification, identification: in image recognition [20, 55], speech recognition [8, 14, 24], text to speech 
conversion [59], signal processing [23, 62], associative memory [2, 18, 19, 49] 

- automatic diagnosis: for expert systems [37] 

- data compression: [11] | 

- games: (38, 44, 63] 

- planification: in robotics [29], in finance {50] 


- function synthesis : [20, 35} 


The technique which is the most widely used today for supervised learning is the so-called “Gradient Back 
Propagation” algorithm (or GBP). The general idea is to modify weights so as to decrease a cost function 


which, for example, could be: 


CW) = Zk I Se Yell? | (1) 
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measuring the discrepancy between the desired output y, and the output S, actually computed by the 
network for the different examples x,. The weights are then modified by: 


AWi,=- HAC/o Win (2) 


This supervised learning model, as the Ap.p reinforcement mode! [5], learns by optimizing an evaluation 


function through a gradient following techniqua. We will fully develop the GBP method in section 4-1. 


After learning, each element in the network will implement a function (depending on its connection weights) 
which is characteristic of the task. We will see that, in the case of multi-layered networks, those functions can 
act as feature detectors, i.e. they implement an elementary descriptor of the task. We discuss this particular 


point in section 5. 

Let us discuss the position of network learning with respect to the Machine Learning approach. 
2-4 Network learning and Machine learning 

tet us first stress the differences, by stating what the network learning approach is and is not. 


- it is not a conceptual but a numerical learning technique: the examples are presented before 
conceptualization, at a very low level of description. For example, the examples could be pixels in an image, 
before any processing of the image, or FFT of a speech sound. In a symbolic approach, the image or the 
sound would be represented as a set of descriptive features. Of course some network models can also use 


preprocessed data (see 5-3-6 for example), but it is not necessary. 


As such, network learning can be compared to statistical methods: there are in fact many relations between 
the two approaches. It has been argued that the statistical approach was not convenient for learning 
purposes because the decisions it provided are not easily interpreted [31]. We will show that the GBP 
method is capable of providing representations which can be interpreted as elementary descriptors or 
concepts: network models, although working in a numerical -or sub-symbolic [60,61] - representation 
tramework, may provide symbolic descriptors. This is one of the main reasons which lead to prefer the GBP 


approach, rather than the non supervised methods mentioned in section 2-3. 


- this technique does not look for explanations, but merely for similarities. Those similarities are not 
conceptual, but purely numerical and represented in an evaluation function which decreases during the 
learning process. Note that, in contrast to some classical Machine Learning methods [41,42], the form of this 
evaluation function is constant, and not improving during the learning process: it is its value which improves. 
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The Network Learning approach can thus be viewed as a fow-level jearning tool by which “raw” data are 
processed so as to generate elementary descriptors and concepts characteristic of the examples. The 
Machine Learning approach would then be a high-level jearning tool, where concepts and symbolic 
representations are manipulated so as to obtain generalizations, explanations. This is summarized in figure 5. 





raw data (noise) 


NETWORK LEARNING 
teaming with examples 
¢superisedieaming 


ENVIRONMENT 
ee: 








"LOW LEVEL" LEARNING 





REPRESENTATIONS 


4 descriptors, rules... 


MACHINE LEARNING 
sinduction, deduction, explanations 
non superisedleaming 
FIGURE 5: learning. 


The figure shows the different levels of learning: starting with raw data, a numerical system progressively produces 


descriptors and rules to represent the structure of the task, which are then used by a symbolic systam. Of course the 


various steps in this process should interact. 


A learning problem couid then be solved by first using a network to preprocess the data, hopefully providing 
an “intelligent” representation and then using that coding to solve the learning problem. This approach has 
already been advocated within the network learning paradigm: successive networks are used to achieve 
tasks of increasing complexity, first starting by the coding task [69]. However, to our knowledge, nothing has 
been done to “interface” a first network step with a subsequent symbolic approach. The problem here is a 
language problem: in one paradigm the representations are numerical, and in the other symbolic. This could 
be compared to the problem of de-compiling machine code into high level languages. Work remains to be 
done in this area, but the reward could be quite important. In particular it could be instrumental in solving the 


painful and costly knowledge acquisition problem in expert systems. 
2-5 Why hidden units? 


The general problem in supervised learning with networks is to realize an association between examples and 
desired responses, i.e. a mapping f: {example} — {desired response}. If this problem is to be solved by a 


network as shown in figure 4, f has to be decomposable into successive functions implemented at the 
intermediate levels (so-called “ hidden ” layers) between the input and the output layers. The theoretical 
question is then about the possibility of decomposing a given function f. The answer to this problem was 


provided by Kolmogorov [33] ina general framework. 


Kolmogorov theorem: Let n2 2 be given, then there exists a family of real functions (Poa)p.q° defined on 


[0,1], continuous, monotone increasing such that for any continuous real function f defined on [0,1}" there 
will exist a family of continuous real functions (Gq)q ; defined on [0,1}", such that: 
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2n+t n 
fx)=D gh Pog (Xp) ] (3) 
qz=1 p=1 


FIGURE 6: Kolmogorov theorem 
The figure shows how a function f in n variables can be computed on a 3-layers network, where the hidden layer has 


n(2n+1) fixed functions Yq and the output layer 2n+1 functions gq which must be adapted to the particular function f. 


This theorem means that any function f in n variables defined on [0,1]" can be computed by a ..iciwork (fig.6) 
with 3 layers: one for the n inputs, one ° hidden " layer with n(2n+1) fixed mappings Pog and one output 
layer with the 2n+1 functions gg. Note that, in this network, the functions ‘Pog are independent of f {only 
functions gq have to be adapted to f) and that the functions Pog only depend on 1 variable each. This 
theorem tells us that indeed we can compute mappings on networks (this is well known by people in 
electronics), but unfortunately it does not tell us how to compute those functions ‘Pog and Qa . The learning 
technique introduced in section 2-3 is one way of learning such functions, but usually, we will accept 
functions Ypq depending on more than just 1 variable, which will allow us to use less than of the order of n@ 
functions in the hidden layer, usually also at the expense of the “ universality ” of those Pg . The idea then 
in those methods will be to automatically derive functions br in the hidden layer, appropriate for the task, 
i.e. implementing good descriptors. The general complexity problem of how many such functions are 
required for a given task is still unsolved. We will see that usually much less than O( n@ ) are necessary. 


This theorem provides us with a theoretical basis to use networks with more than just 2 layers (one for the 
inputs, one for the output). The first adaptive systems were based on networks without hidden layer, and 
this was an intrinsic limitation which prevented them to solve “hard” problems [18,58]. 


3- AUTOMATA NETWORKS 


Connectionist models are based on the use of automata networks, i.e. large sets of many interconnected 
simple elements. It is their dynamical behavior that allow to memorize and retrieve information. We will briefly 


present here what we mean by automata network. 


3 
3-1 Definitions 


An automaton is an element defined by an internal state and a transition function f which allows it to 
change its state. The state space S might be {0,1}, where 0 would be interpreted as inactive and 1 as active or 
the two grey levels in an image. S might also be {0,1 ,..4k} for an image in k+1 grey levels or even a continuum 


[-4,4]. 


{ depends upon the states of the connected automata. The automaton, in state s € S , receiving signals sy, 


..« Sp,» Will switch to state s' computed by: 
s' = f(S, $4, ---. Sp) (4) 
Example: an automaton realizes the AND function, if it receives two signals sq and So and: 


si= 1 f S$; =So=1 (5) 


0 otherwise 
An automata network is a set of interconnected automata. Each automaton receives signals only from the 


automata to which it is connected (or from the environment). In the following, we will always assume that all 


automata have the same state space S and that the signals sent are just the states of the automata. 


3-2 Different classes of automata 


In the following, we will use mainly linear-type automata. The transition function fj of such an automaton i 


is a function, eventually non linear, of the total input Aj to the automaton: 
si =f (Ai) with = Aj= Xb Wih Sh | (6) 
where Wip is called the connection welght of the connection from h to i. 


When { is the identity function, and S C IR the automaton is called IInear (fig.7). In paratlel mode {18], the 
dynamics of the network is thus defined by: 


$(t+1) = W sit) (7) 


where W = (Win }jp, is the weight matrix. 
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FIGURE 7: Linear automaton 


The figure shows an automaton with internal state sj, receiving connections, labelled Wi1,-.-.Win from other automata in 
states $4,....89. The automaton updates its internal state by computing the weighted sum of its inputs Aj = Xb Wih Sh 


and outputs this new state sj = Aj. 


if tis a step function and S = {0,1} or S = {-1,+1}, then the automaton is called a threshold automaton -with 0 


threshold- (fig.8 -1 et 8-2): 


sj= 1 if Aj 20 or 1 if Aj20 (8) 
0 otherwise -1 therwise 
which we denote: sj = i[Aj] (9) 





(1) (2) 


FIGURE 8: different classes of automata. 
The figure shows a 0-threshold automaton with state space S={0,1}(1)andS={-1,+1 } (2), a saturation automaton 


in the case where $c R (3), and a smooth automaton (4), for T varying from 4 downto 1 (the steepest curve): the 


threshold case corresponds to T=0. 
Note that, more generally, an automaton with transition function: 


is called a threshold automaton with threshold 6}. A threshold automaton with threshold Bj can be 


considered as a threshold automaton with 0-threshold by adding an auxiliary automaton in the network, 
tabelled 0 for example, which state will be kept constant in -1: B; is then the weight of the connection from this 


automaton n°0 to automaton i (fig.9). In the following, we will usually use 0-threshold automata, and possibly 


add this auxiliary cell to the network. 
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FIGURE 98: 


The figure shows how it is possible, by adding an auxiliary cell held constant in state -1, to replace any threshold 


automaton with threshold 8; by a 0-threshold automaton. 


Threshold automata were introduced by McCulloch and Pitts [40] as models for the nervous cells: the 
neurons. Neurons are cells which, when activated, fire, i.e. propagate a signal along their axons to connected 
cells. This activation is triggered whenever the weighted sum of the signals received passes an activation 
threshold: this is similar to equation (9'). Because of this connection with neurons, automata networks are 
often called neural networks. Recently, work in statistical physics [27] has shown the formal analogy 
between threshold automata and spin glasses. This analogy has proved useful to establish some properties 
of the networks, especially through the use of an energy function {20, 27]. Such an energy function was also 


derived in the context of Adaptive resonance theory [10]. 


Multi step threshold automata (also called saturation automata) are used for example in the BSB mode! 


(Brain State in the Box [2]). They are defined by: 


sj = SATUR[ Aj] (10) 
with SATUR[u]= uU tt ue [b,B] 
b f usb 
B f# u2B 


where the state space is the “box” S =[b, B] (fig. 8-3). 
Finally, in the case where S = IR, we will often use a sigmoid function f: 
f(x) =afekU-1]/ [el +1] (11) 


The parameter T = 1/k is called the temperature by analogy with the spin glasses. Varying T from high 


value to 0 will transform a smooth automaton into a threshold automaton (fig.8-4). 


We will see in the following that these automata have different properties for leaming tasks. 
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4- MULTILAYERED NETWORKS 


4-1 The retro-propagation algorithm 


All the early learning networks were either made of just one automaton (adaline [67]) or had only one layer of 
adaptive weights (perceptron [43]). This is why they were intrinsically limited to “easy” problems (i.e. linearly 
separable). We have seen, with Kolmogorov's theorem (section 2-5) that a network with 3 layers could 
compute any real continuous function in n variables over [0,1]. However, this theoretical result was not 
known to the designers of the early systems and it took a long time, following the perceptron " failure’, before 


this idea of "hidden units” was used. 


The idea of using networks with “hidden units”, with no direct interaction with the environment, was first 
introduced by Hinton and Sejnowski for the Boltzmann Machine [1]. Their machine was made of stochastic 
elements (see [26]). They showed that it was possible to learn with this machine problems which were not 
learnable by 2- layers machines such as the perceptron [57, 58]. Their work thus opened the field for multi- 


layered networks learning. 


Multi-layered networks of deterministic elements were then introduced: their structure is quite similar to the 
perceptron. They have one layer receiving the inputs (the retina), one layer which broadcasts the output and 
one -or more- hidden layers in between . The layers are interconnected from input to output layers with no 


backwards connection allowed (fig.10). 


layer 0 (input) n-1 n n+1 N (output) 





hidden layers. 


FIGURE 10: multi-layered network 
The figure shows a network with N+1 layers. Layer 0 receives the input patterns xy. The signals are then propagated 


forward through the network (there are no backwards connections), and the hidden layers successively compute their 


state, until the output layer, where the computed output S; is compared to the desired output yx. 


This model can be viewed either as a generalization of the perceptron or of the adaline. In the first case, the 
learning algorithm propagates desired states: this was the idea developed by Y. Le Cun in his HLM algorithm 
[36) (HLM stands for ° Hierarchical Learning Machine "). In the second case, the learning algorithm 


17 


propagates an error signal: this is the GBP algorithm (or “ Gradient Back-Propagation ” ) proposed by 
Rumelhart [52]. Other authors proposed similar rules 145, 46]. . 


What makes the main difference with the perceptron rule is that all connection weights are modifiable during 
learning: in particular the hidden units are modified, whereas the association units were not. This will allow a 
network to progressively derive its appropriate set of feature detectors, by using the regularities it discovers 


in the examples, instead of being programmed to du Lo. 


4-2 Retro-propagation of desired states 


We briefly present here the HLM mode! [36]. It ts based on an automata network with binary units, i.e. cell i 


computes its output by xj (either +1 or -1): 
xj= 1 [Ai] where Aj=2h Wih Xh (12) 
The idea [36] is to compute, for each celli, its desired state dj: 


- for the cells which are on the output layer, this is easy: dj must be equal to the desired response yxj (when 
input x, is presented). 

- for a cell iin a hidden layer, let us suppose that all the cells h to which i sends its output have their desired 
state dh already known. Then qj will be chosen so as to best “satisty "the downstream cells h. 

For example, suppose i is connected to only one cell h and that Wpi > 0. Then it is clear that dj will have to be 
equal to dp, which will allow h to produce its desired output dp. i Wj <0, then we must have: dj = - dh. 


If cell iis connected to many celis, then a compromise wil! have to be found to satisfy as much as possible all 
the downstream cells: if Whi_ is large (in module), then the contribution of dp, to dj will be large. 


Many different algorithms have been proposed along those ideas [37]. We present here one of them, which 


was implemented for a task of character recognition [36]: 


dj = yki if iis on the output layer (13) 
dj =1{ Xp Whi dh) it iis in a hidden layer 


The computation of the desired states is thus quite similar to the computation of the states {equation (12)) 
except that the connections are used backwards, from the output layer to the input layer. 
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The weights are then modified with a rule similar to a Widrow Hoff rule: 
Wih (t+1) = Win (t) +H (dj - Ai) Xh (14) 


This algorithm can solve “hard” problems, such as the recognition of patterns with translation invariance [36}, 
unfortunately it is relatively unstable and also requires more cells (and thus weights) than the GBP presented 
below. However, the computations involved are not as heavy as for the GBP, since it only involves binary 


variables. 


Plaut & al. [47] propose a modification of the HLM procedure, which uses 4 criticality factor pj for each cell: pj 


will be 1 if all downstream cells “ agree ” and 0 if they are completely contradictory. Weights will be modified in 
proportion with pj. Equations (13) and (14) are then modified by: 


dj = Yki H iis on the output layer (13°) 
di=4[ 2h Whi dh Phi if tis in a hidden layer 

pi = | Dh Whi ShPhl/ Zh | Whi dh Pht (15) 
Win (t+1) = Win (t)- Pi (dj - Xi) Xh (14°) 


The authors report reduced computing time with respect to GBP but with less stability. In fact, GBP seems to 
work better than any other method proposed sofar, we Now describe ft. 


4-3 Retro-propagation of error signals 


The GBP algorithm works as follows: each element i in the network has an internal state xj which can be 


updated depending on the signals it receives from the elements interconnected to it: 
xj = f [Aj] with Aj = Lh Win Xh (16) 
where Win is the weight of the connection from h to i (fig.10), and fis a sigmoid function (fig.8). 
The network is me to realize the associations x, — Y,, k = 1...m. When the network receives an input x,, 


it propagates this state through the network using rule (12). The state S; of its last layer is called the 
computed output associated to x,. It is compared to the desired output y;, and the associated cost is: 


Cre It Sk - Yk II? = Ls [Sks - Yks J? — (17) 
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where s indices the units in the last layer. The total cost is then: 

C(W) = 1/m Dy Cy (18) 
C is minimized by a Widrow Hoff rule: 

Win (k) = Wij; (x1) - B (kK) 8 Ox / Win (19) 
But: 0 Cy/ Win = 0 Cy/0 Aj. Xh (20) 


Hence, the @ C; / dWjp are computed by retro-propagating from the last to the first layer the error signal (for 


convenience in notation, we drop index k in fj ,): 
fi =O JAA 


It is easy to show by simple computations (mainly chain rule: see for example [20, 26]) that: 


Fi= 20 Si - Yui) f(A) ifiis acellonthe output layer (21) 
fi= [Zh Whi (k) . fh]. PCAD otherwise 


Equation (27) thus becomes: 
Win (k) = Win (k-1) - 2K) fi. Xh (22) 
The GBP algorithm proceeds as follows (fig. 11): 
1- at time t, present an input pattern x, on the input layer. 
2- propagate the states forward up to the output layer, by using equation (16) 


3- compare the computed and desired output S_ and y_ and derive the error signa! fj on the output layer. 
4- retro-propagate the error signa! fj from the output layer to the first layer through equation (21). 


5- modify the weights by equation (22) and back to step 1. 


computation Xp, X; Xm 
of states x 7 7 7 } 


, / / computation of 


f f f error signals f 
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FIGURE 11: the GBP algorithm 


The figure shows one pass of the GBP algorithm: states x are first propagated forward and then error signals f are 


propagated backwards, to allow modification of the weights Whi- 
Sometimes a modified rule (with “momentum” [52]) is also used: 

Win (k) = Win (k-1) - 4 Win (kK) (23) 
with: A Win (kK) = (WK) fi. Xn + BA Win (-1) (24) 
It improves the stability of the convergence process. 
4.4 The fInks between binary and continuous models 


We now summarize the different automata networks which can be used for supervised learning (fig. 12): 


either binary (eventually with noise) or continuous (either linear or smooth). 














perceptron 
Binary unit adaline(LMS) Real linear unit 
pseudo-inverse 






f=Identity 


Smooth unit with function f 


states 
GBP HLM 








» 


Binary unit with noise nf 


reinforcementieaming 
FIGURE 12: network learning. 


The figure shows the relationship between different mode!s for network learning. 


Early network models were based on binary units (perceptron and adaline). Linear models of associative 
memory [32] use networks of continuous units with linear transfer functions, and a pseudo-inverse method 
for learning. Binary units with noise are used in reinforcement leaning models (see section2-3). Smooth units 
networks have learning algorithms which back-propagate either desired states (HLM) or error signals (GBP). 
In fact, one can show that smooth units networks and networks of binary units with noise are “equivaient”. Let 
i be a binary unit with noise 1): 


y= 1 ff Ap+ N>0 (25) 


0 otherwise 


Then, it is easy to show [3, 68] that: 
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Pr(x;=1)=Pr( Aj + T1>0)=1- WY (- Aj) 


and thus the expected value of the random variable x; is : 
E(x,)=f(Aj) (26) 


where { is defined by: f (x) = 1 - ‘¥ (-x) (27) 


Hence a binary unit with noise 1] behaves on average like a smooth unit with transfer function f defined by 
equation (27). It is also possible to simulate with such a noisy binary unit a smooth unit following a GBP 


algorithm {3}. 


5- THE PROBLEM OF GENERALIZATION 


The crucial problem in learning systems is of course their capability of “good” generalization. In this section 
we will discuss what is meant by that with network models and also, what exactly are the available results. We 
will argue as Laird & al. put it [34] that: «One of the most important sources of generality is the representation 
used for the task states» (p56) and thus will show, when it is possible, how the network model implement 


such representations. 
5-1 What does It mean to generalize for a network? 


First, one must acknowledge that generalization is an ill-posed problem. For instance in a supervised 
method, a network has to fill in a table (fig. 13) where only some lines are given through the examples and the 


associated desired responses. 


Examples Inputs Outputs 





FIGURE 13: generalization tet 


The figure shows a table, representing the values of a given mapping (the outputs) for different values of the inputs: all 
variables here are assumed to be discrete. Examples are presented to the network, i.e. some lines of the table are 
specified. The learning procedure specifies the weights of the network, and thus fills-in the remaining lines of the table. 


The network can then be tested by presenting new inputs and looking for the output produced by the network. 
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The task of the network is such that presented with an input never seen before it has to choose a satisfying 
answer , i.e. coherent with the examples. Of course, “satisfying ” is a notion relative to the user who has an a- 
priori knowledge of the task. If the table of figure 13 is to be filled-in correctly, then the user must give to the 
network some hint about the “ semantics ” of the problem. Stated in other words, there is no free meal: 
suppose we want to teach a network to recognize letters. We might like it to “ generalize” by eliminating 
noise or by allowing for translation invariance for example. If we do not “tell” the network which of thase two 
tasks we want it to solve, there is no way by which it could gi evs. But, the nice thing with networks is that you 
can tell them in an implicit way: instead of giving them a model of noise in terms of distributions with specific 
parameters or rules, you can just show them with some examples of noisy data (with this distribution). 


Examples foliow below . 


Remark also that one way to fill-in the lines of the previous table could just be by using a metrics: the output 
to a new input would be the output associated to the closest (in Hamming distance) input. Hamming metrics 
can be interesting, but very often, the semantics of the problem cannot be characterized by such a metrics: 
for example two different “A” can be easily recognized, even though their Hamming distance is large. We 
thus need systems which generalize by using a metrics which include some knowlledge of the semantics of 


the task. 
5-2 Conditions for good generalization. 


For a network, learning consists in organizing the examples space around clusters. These Clusters are buitt 
by sampling space with the examples. Of course, if sampling is not adequate in some clusters during the 
learning process, there will be little hope to correctly generalize to test-patterns in those Clusters. This means 
that the performances of the learning system heavily rely on the quality of the sampling. This is well known to 


everybody in the Machine Learning community and we will not insist on that. 


Usually performances increase with the size of the learning set, but this is not always the case: if the task has 
no intrinsic regularity, giving the network more examples will not help, it will just add more confusion (section 
5-3-2). In that case, we show that the network has devised exactly the rules we would have used ourselves. 


We mentioned before that good generalization was obtained if a good representation is generated during 
the learning process. This can be seen in experiments of incremental learning where broad indications of the 
task are first presented, which induces the network to derive preliminary representation. Then, providing it 
with more refined examples of the task makes it further refine its representation set, but often, even with the 
crude representation available, the system is able to solve some of the finer task. This is illustrated in section 


§-3-2. 


a al s owen 
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One nice feature of the GBP method is that it gives indications on the learning set. If the performances on 
one cluster are far worse than on others, that points to a possible bad sampling of this cluster during training. 


This is shown in section 5-3- 5. 


Finally, preliminary knowledge can always be used to guide the generation of good representations. By 
giving the network an architecture which depends on the task, one can force the network into elaborating 


representations more appropriate for the task. This is shown in section 5-3-6. 


5-3 Is generalization possible? The proof Is In the pudding. 


in order to discuss the generalization capabilities of the GBP algorithm (exclusively used here), we now 
present a few experiments hich were designed to illustrate some of the critical points about generalization. 
Those experiments will be described rather rapidly, the interested reader can refer to the corresponding 


publications for more details. 
5-3-1 Are multi-layered networks superior? Associative memory 


We have tested muttilayered networks against 2 - layers networks on various tasks of associative memory (fig. 
14). We show here the results [19] for the letters of the alphabet (fig. 14). The units in both networks are 


identical, they follow a smooth function as given by equation (11). 
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FIGURE 14: learning the alphabet 
The figure shows two networks, with 2 (left) and 3 (middie) layers, which have been used to learn the letters of the 


alphabet (right). Their input and output layers are 8x8 grids (right) where the letters are digitized and the 3-layers network 


has a hidden layer with 32 cells. With full connections among layers, both networks have the same number of weights. 


We wanted the network to learn to eliminate noise. We thus compared the performances obtained when a 
model of noise was given and when no model was given. Both networks were trained with the same GBP 
procedure. In one case, only the * pure ” patterns (the letters ) were presented. In the “learning with noise” 
experiment, the examples were patterns which were obtained from the letters by inverting 5 bits at random, 


the pure examples were also presented. 
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FIGURE 15: learning with nolse 


The figure shows the recognition rate as a function of the noise level in the input. Clearly, the 3 - layers network is better 
than the 2 - layers network, when noise is used during learning. However, this is not true when no noise model was 


provided. Here we have memorized the 18 first letters of the alphabet. 


The results are shown in figure 15. Performances decrease when more noise is inserted in the input, the 3 - 
layers network performs very well with respect to the 2 - layers network, when noise has been * shown”. In 
the “ no noise ” experiment, the 3 - layers network performs rather poorly: this is due to the fact that, in that 
case, the 2 - layers network is related to the pseudo-inverse model [32], which is known to have a “built-in” 


model of noise: namely gaussian uncorrelated noise [20}. 


We have also tested different network architectures on a task of noise elimination for auto association and 
hetero association of random patterns [21]. An example here is a random vector with 32 bits. Networks with 
2,3 and 4 layers have been used. The network with 3 layers has 16 hidden units and the network with 4 
layers has two hidden layers of 13 cells each. The 3 networks have approximately the same number of 
weights. The networks were taught in noisy environment: each example was presented together with noisy 
patterns at distance r from it (r was varied from 6 down to 0). This task doesn't present any obvious reguiarity, 
however the networks can learn to realize it. Correspondingly, the task of hetero-association (i.e. associating 
with a random vector another random vector) has been found harder for the network, but it could 


nevertheless be realized and even provide noise elimination (fig. 16). 
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FIGURE 16: learning random associations. 
The figure shows the performances of the various networks for auto-association (left) and hetero-association (right) of 


random vectors. Performances degrade with an increasing number of memorized patterns and are better for networks 


with more hidden layers. 


The networks also exhibited an interesting feature: each sweep of the learning set contains one example of 
each “pure ” pattern, and various examples of more and more noisy patterns. The t.etworks were found to 
first learn the easy task: recognize correctly the pure pattern and then learn, with more time allowed, the 
harder task of recognizing the noisy patterns . This is probably due to the fact that the network progressively 
derived interna! representations increasingly adapted to the noise elimination task. Of course, in this case, 
the internal representations computed by the network did not yield to analysis. 
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FIGURE 17: learning easy things first 


The figure shows the performances of the 4 - layers network, during the learning process: at the beginning, the network 


learns to correctly recognize the pure patterns, and progressively builds up a model of noise. 


Learning with noise has also proved useful in different contexts, such as speech recognition for example 


[14], where performances were improved when noisy versions of the examples were shown to the network. 


5-3-2 Devising good intemal representation: learning * topology ” [21]. 


The problem is the following: suppose we have a“ signal” , represented here as a succession of n bits, 
where we want to detect the presence of a * pattern” : here the pattern will be a sequence of at least 3 
successive cells in state 1 (we call this problem the 3 - in- n problem). This is somewhat related to a problem 


studied in [56]. Figure 18 shows examples and counter examples of such pattems. 
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The figure shows a network (left) with 3 layers capable to learn to recognize examples where a pattern is present in a 


signal (middie) from examples where it is not (right). 


There are 256 possible “signals ”, i.e. 8bits sequences: a Subset of those was used for training ,this was the 
learning set, which was usually formed of about 50% “examples” and 50% “counter examples ”). The 
remaining signals were used as test set: after the learning session, items in the test set were presented to 
the network and the answer of the network checked against the correct - known- answer. This gave the 


accuracy of the generalization. 


We have used different networks with 3 layers and an increasing number of hidden units in the hidden layer. 
We observed that the performances increase with more hidden units, but that the generalization can get 
worse (fig. 19). We thus further used networks with just 3 hidden units, an arrangement that revealed 


sufficient for both good learning and generalization abilities. 





> 1 g 
E 39 
Fe g — nonrandom 
2 : <+ random 

8 

he; —@ learning a 

generalisation 
6 ” 
: oO 50 100 150 200 250 
0 2 4 6 8 10 12 size of the leaming set 


nb hidden units 
FIGURE 19: solving the 3 - In - 8 problem. 
The figure shows how the performances, both on the learning set and on the test set, vary when the number of hidden 
units increases (left). When the size of the learning set increases, the performances - on the learning set - improve, if 
there are sufficient regularities in the task to be learnt. On the other hand, if the task is “random”, giving more examples 


will do more harm than good(right). 


When one increases the size of the learning set, one usually expects to get better results since more 
“information ” is given to the network. However, this is only true if there are inherent regularities in the task 
that the network can build upon. If the task is random (fig. 19 right), i.e. if signals are associated to random 
answers during training, then giving more examples to the network will not help it: on the opposite, 


performances will degrade, with an increasing size of the leaming set. 


Now the question can be asked as how the network was capable of solving this task. The problem clearly 
involves some knowledge of topology, since it amounts to detecting whether a block of adjacent cells are 
active. One way to give that knowledge to the network would have been to provide it with an appropriate 
architecture: if the hidden units compute an “AND” of 3 successive cells (a “window”) and if the windows 
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associated to successive hidden units overlap, then an “OR” on the last layer will solve the problem. 
Therefore giving local connections from the input layer to the hidden layer would have helped the network 
solving the problem (which was indeed the case with such an architecture). However, with the architecture 
given in figure 18, the network did not receive any information about topology. Somehow, it derived such a 
notion and this can be seen in the internal representations computed by the network. Figure 20 shows the 
weights of the 3 hidden units in a 3 - layers network, for an increasing learning set size. It is quite clear that 
the network has devised a system of “ windows ” with which it checks wheth2r the pattern is present: in the 
case of 3 hidden units, the windows are 4 bits long and overlap on 2 bits, which is a very neat way to solve 
the problem, since all cells are in fact performing the same task (which respects to the symmetry of the 


problem). 


The internal representations here are easy to understand in terms of the task they are performing: they can 
be characterized by a boolean mapping - or a tule - on the input (precisely the 3-in-4 rule) followed by an OR 
on the fast layer. Networks with more than 3 hidden units devised more or less the same sort of strategy as 
long as the number of hidden units was not too large, afterwards, the representation was not so easy to 


analyze. 


It thus looks like there would be an optimal number of hidden units (i.e. of parameters) for which the learning 
and generalization accuracies are correct and the representations possible to interpret. With larger numbers 
of hidden units, the generalization remains the same but the representation is difficult to interpret. 
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FIGURE 20: Internal representations. 

The figure shows the weights devised by a 3 - layers network with 3 hidden units. The top 2 curves give the 
representations when the learning set had 50 and 100 patterns. The bottom curve (left) gives the * perfect” 
representation obtained when all 256 signals were used for training. Clearly, when provided with smaller sized learning 
sets, the network tried to get to this same representation: in fact 3/5 of the 256 signals were sufficient to correctly 
solve the problem by this method. The bottom curve (right) shows the weights when the network was trained on a 


“random” task with 256 signals: clearly the hidden units are not doing anything easy to interpret. 


5-3-3 A semantic netwom. 


In these experiments, a network has been taught to memorize a semantic network; no effort has been made 
to implement a network with an * interesting ° structure, as in [26]. Rather, the general memorization ability 
was investigated together with the internal representations generated ( simulations have been run by S. 


Abadi and Moureaux-Nery). 


Part of the semantic network is shown in figure 21, it has 22 nodes and 5 relations. A network with 22 + 5 
nodes for input and 22 nodes for output is used and the learning session consists in presenting examples of 


the form: 
OBJECT 1 RELATION (input) -—> OBJECT 2 (output) 


such as for example: human hunts animal , Paul loves Mary ... 
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FIGURE 21: semantic network. 
The figure shows part of the semantic network which was memorized (left) by the network, shown right: the network has 


22 + 5 nodes for input, 9 hidden units and 22 nodes for output. 
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The network is relatively long to train, but eventually succeeds in memorizing all the examples. Its internal 
representations are then showed in figure 22. They are relatively hard to understand since those 


representations are distributed: one cell usually codes for various features. 
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FIGURE 22: Internal representations 


The figure shows the internal representations devised by the network: the columns give the activity levels of the 9 


hidden units when shown the different examples. 


Such a network could be used to answer questions about the semantic network, or to find a common 


ancestor in a taxonomy (by re-iterating the process). 
5-3-4 Incremental learning: Multi font character recognition [55] 


Characters in various fonts have been acquired by camera and digitized, then coded through classical 
techniques in image processing: squelettization, line detection... into 5x20 patterns. Those patterns were 
used to teach a network to recognize the corresponding letter (only the capital letters were used) (fig. 23). 
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FIGURE 23: multi font character recognition 


The figure shows (left) the network used for muhi font character recognition, some hidden units are locally connected to 
lines, columns, windows in the input pattern and others are fully connected. On the right, we show the incremental 
learning performed by a network learning more and more fonts. Learning at the beginning is slow, since the network has 


to derive from scratch its knowledge about fonts. Later on, the network can build upon previous knowledge to fit in a new 


font. 
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The network is first shown with 3 different fonts, after about 60 sweeps of the learning set, the fonts are fully 
recognized (fig. 23). Then, 2 new fonts are presented to the network, they are not completely recognized 
(about 35% errors), but after only 20 sweeps the network recognizes them perfectly. The whole set of 5 
fonts is then represented (10 sweeps) and more and more fonts are progressively added this way to the 


network. 


it seems that the first few fonts are sufficient for the network to “understand” what a font is and perform 
relatively well on new fonts. Very little additional learning is necessary to add a new font to the network. 


The internal representations of the network [55] are constructed from the “masks” originally wired into the 
architecture of the network. In fact, these representations are refinements of those masks adapted for the 
fonts provided to the network: they constitute a basis of features for those fonts. The output unit 
corresponding to any given letter (let say a “C”) receives connections from those features which are 
significant for that letter: for instance, “C” would have connections from internal units coding for horizontals in 


appropriate positions and masks for vertical in the left. 


Learning thus proceeds here by refinement on a set of gross feature detectors provided in the architecture 
of the network: those detectors “hand-wired” reflect the a-priori knowledge about the task. 


5-3-5 Testing the learning set: leaming NMR spectra 


The problem here is to distinguish between two classes of chemicals by means of the NMR spectra of C13. 
Class 1 corresponds to the formula: X - CH2 - CH2- Y and class 2 to: Y -X-CH2 - CH3 

The frequency rays of the two carbons (a and b) are measured and coded on 7 bits. Each example is thus 
characterized by 14 bits, corresponding to the 2 frequency rays of its C. This problem has been solved by 


symbolic methods [6]. 


The network shown in figure 24 is first presented with a learning set and then tested with a test set 
(simulations have been run by C.Rouveirol and L. Tcheeko). The curve shown in figure 24 gives the results 
for two leaming sets of sizes 34 and 24 respectively (both with 50% items in class 1 and 50% in class 2) and 
various test sets. Test set n°1 has 34 patterns, 18 in class 1 and 16 in class 2. Test set n° 2 has 21 patterns, 
14 in class 1 and 7 inclass 2. Test set n°3 has 14 patterns, all in class 1. The results shown here indicate that 
the generalization performances are clearly better for items in class 1. Items in class 2 are very badly 
recognized, which must imply that the learning set did not represent correctly class 2. This can be viewed as 


an a posteriori testing of the learning set (in the case where the test set is known!). 
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FIGURE 24: NMR spectra 


The figure shows the network used and the results of learning and generalization with 2 different learning sets and 3 test 


sets. Clearly class 2 was not correctly learnt. 


5-3-6 Role of the architecture: Automatic diagnosis [37] 


Several design strategies can be used when faced witha real-world problem. One is to use “brute force”, i.e. 
to choose a very large network with no (or little) specific structure and to hope that the learning algorithm will 
capture the regularities and will generalize correctly. However, this strategy is not realistic for large problems. 
In most cases, it appears to be useful to guide the generalization process. This leads to a secoru possible 
strategy which is to put some knowledge about the task into the architecture of the network. 


We have tried to apply this technique on a medical diagnosis problem (this work was done in collaboration 
with J.L. Golmard INSERM, CHU Pitié, Paris) 


The examples used for learning come from a large database containing 5767 cases of abdominal pain. 
Among these cases, 4027 are used for the training set, and 1740 for the test set. 


Each case is described by 132 qualitative or semi-quantitative symptoms, together with the (supposedly 
correct) diagnosis. There are 21 possible diseases (appendicitis, cancer, abortion, hernia...) and an 
-additionnal diagnosis calied “non-specific pain”. This last class is highly inhomogeneous since ft contains 
the “non-disease” cases, and some diseases which do not need to be cured. More than 24% of the cases 


are non-specific pains, and nearly 27% are appendicitis. 


The descriptions are coded into 316 binary descriptors. Each qualitative symptom uses two bits: the first bit 
is on when the symptom is present, the second bit is on when the symptom is not present, both are off when 
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the information relative to the symptom is not available. Quantitative symptoms use a coding by position: as 
many bits as there are possible values. An average of 80% of the symptoms are known. 


symptoms 316 62 
diagnosis syndroms 
=<? =? —_ — 8 -- + 
22 2222 —> 22 22 
316 316 —> full connection 


sven > local connection 


FIGURE 25: networks for medical diagnosis 
The figure shows various architectures which were used for the medical diagnosis problem. The network which included 
“syndrom” cells (right) led to the best results. The layer in the right of this network and the one in the middle of the figure 


were added after some learning had already taken place: the connection weights were initially set to the identity 


mapping and further modified through the GBP rule. 


The network (fig. 25) has 316 input units, 22 output units, and a variable number of hidden units in one or 
two hidden layers. The first hidden layer is divided into 12 groups. Each group is labeled by a “high-level 
diagnosis” or syndrom, i.€. a general class of diseases (urinary disease, inflammation, obstruction of the 
bowels, peritonitis, gynecological disease,...). Two of these groups are attached to special tasks: one for 
processing the pain localization, and the other one for helping discriminate appendicitis from non-specific 


pain. Among all the possible connections only the fotiowing are allowed: 


- from symptoms to syndroms 
- from syndroms to diagnosis 


- from symptoms to diagnosis 


inside each of these groups, a connection is made only if there exists a causal relation between the source 
concept and the target concept. For example, we put a connection between the syndrom “gastric or 
duodenal! disease” and the diagnosis “ulcer” because an ulcer causing abdominal! pains is generally situated 


in the stomach or the duodenum. 


This method allows to reduce the number of weights, and generally improves the generalization 


performance without decreasing the performances on the training set. 


The resulting network contains 3380 weights and 62 hidden units. After a first training session of about 10 
sweeps through the training set, a new layer is added to the network. This layer has 22 units and is used as 
the output layer, it is fully connected to the previous output layer. The initial weights of these connections 
are set to the identity matrix so that adding this new layer introduces no perturbation. Then 10 more sweeps 
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through the training set are performed. After each presentation, the output units are sorted by activity 
levels. If the right answer is in the three most active units, the output is considered as correct. The 
perfomances are 75.3% on the training set, and 71.3% on the test set. The most frequent confusion is 
“non-specific pain” classified into “appendicitis”, these errors represent 40% of the total number of errors. 


It is too early to draw clearcut conclusions from these results. Our understanding of the back-propagation 
algorithm on large networks and real-world data is too weak. Nevertheless, it is a first step toward a mixing 


between knowledge based systems and neural network approaches. 


6- CONCLUSION 


in this paper, we have described a collection of techniques which were, until recently, conceived, developed 
and advocated outside of the Artificial Intelligence circles. They share some distinctive features that make 
them appropriately belonging to the same cluster of methods: 

- distribution of knowledge and contro! 

- use of low-level symbols: mainly numerical | 

- learning through many examples 

- adaptiveness and robustness to fautts. 


Through the use of very many simple processing elements operating in parallel and organizing themselves, 
they capture some regularities in the tasks to be done. Thence neural networks offer a possibility of new 
approaches in many fields: speech recognition, image processing, medical diagnosis, biology, physics, data 
compression, system architecture, ... Now, VLSI technology has developed enough to make feasible the 
realization of such networks. Massively parallel computation can then be effectively realized, increasing their 
potential for real-worid applications. 

The huge number of connections makes the network very resistant to defaults or partial destructions af its 


ejiements. 


This old and young again approach is now demonstrating its feasability and its maturity through several! 
systems and applications that are far from trivial : character recognition, speech recognition, diagnosis in. 
domains with weak theories or uncertain data, and so on. 

As is apparent in this paper, many interesting models have been developed along these lines. Some of them 
offer competitive alternatives to other well-tried techniques. For instance, Kohonen [32] approaches state- 
of-the-art continuous speech recognition systems, and other realizations [8, 11, 39] reach now comparable 
level of achievement. However, and this is the message that we would like to take across in this paper, it is 
time now to bring together the traditional A! techniques and philosophy, based on the use of high level 
symbols and intensive domain knowledge, and the new connectionist spirit, that comes with robust and 
general learning algorithms suited for the treatment of low level raw data on distributed systems. Both 
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should benefit from mutual recognition and understanding and, even more to the point, from active 
collaboration. 

As in the classical Al methodology, knowledge of the domain has to be put into the system by the expert. 
This appears at three different levels: the coding of the data, the design of the architecture, and the choice of 
the learning algorithm. Note however that coding operates at a much lower level than in standard Al systems. 
No formal methodology exists as yet to determine the correct architecture and algorithm needed to solve a 
given problem, but results obtained so far seem to show the existence of certain universal strutures, able to 
cope with different problems, like graphem-phonem transcription and speech recognition. 
This leads us to sketch here some lines along which we think lie promises and which should be explored in 
details. 
- To try to interpret in terms of high level descriptors the structures spontaneously evolving in neural 
networks when trained on specific tasks. 
The determination of pertinent descriptors remains indeed one of the most difficult tasks, and the less 
mechanized, in the conception of Al systems that must deal with weakly formalized domains, and is 
crucial to the design of knowledge based systems in general. It must be noted that already some first 
encouraging works have been done, for instance [53], with "diabolo" like propagation networks also 
called "identity mapping” [11] (using a small number of hidden units). 
In this perspective, we can envision learning systems integrating connectionist networks at the 
perceptual level with Al generalization and induction systems a la Michalski or a !a Kodratoff to cite but 
two of them, that are more high level symbolic and explanation oriented. 
- To derive rules usable for expert systems from careful study of the structure of the connections in 
networks trained through intensive supervised earning sessions. This could be another way of 
automatically obtaining rules from the "show how” rather than the “know how", since finding pertinent 
examples is a task at which experts are usually notably better than explaining how they reason. 
- To understand how is really implemented the distribution of knowledge representation and how 


distribution of control can be generalized to other Al methods. 


These are just but a few ideas that are mainly intended to boost and stimulate a collaborative approach 


between Al and connectionism. 

The potential for the Machine Learning community seems to be very promising. We hope to have proved that 
an interlace between the two approaches would be both profitable and possible. We think that this will 
undergo a huge development within the next few years and that many more applications will be realized. 
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